Multi-granularity for knowledge distillation

Considering the fact that students have different abilities to understand knowledge imparted by teachers, a multi-granularity distillation mechanism is proposed for transferring more understandable student networks. A self-analyzing module of teacher network designed, which enables learn from teaching patterns. Furthermore, stable excitation scheme robust supervision training. The can be embedded into frameworks, are taken as baselines. Experiments show improves accuracy 0.58% on average and 1.08% in best over baselines, makes its performance superior state-of-the-arts. It also exploited student's ability fine-tuning robustness noisy inputs improved via mechanism. code available at https://github.com/shaoeric/multi-granularity-distillation.